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ABSTRACT 

The practice of analyzing all available project 
children in as large a group as possible is considered not to be 
justifiable when distinct subgroups of pupils are represented. 
Instead, the approach suggested here determines the test score gainfa 
pupil achieves from the beginning to the end of the year, with all of 
the pupil gain scores of a simgle compensatory teacher being compared 
to those presented in the Metropolitan Achievemei^t Test (MAX) Gains 
Tables. These tables pr'^sent typical gains that should be aade on a 
particular subtest of the MAT, according to the grade level of the 
pupil and according to whether the pupil's pre-test achievement level 
is high, average^ or low. Thfese tables differ from the norm. tables 
provided in the test publisher^^s manual in that both the pre and 
post-test were administered to the same pupils to obtain the gain 
scores. Th6 tables* have two major limitations: (1) they are only 
useful for pupils* test results who are in grade levels 2-8; and, (2) 
^the interval between testing is short. Implementation considerations 
and implementation procedures are included in the discussion. *MAT 
Gains Tables are presented in this ^publication. (Author/AM) 



/ ■ 



4t 4t 3|e 3|e ^ 3|e )|t 4( s|e 3|e )^ 3(e ]|c ]|t ]|e 3|e 3(e ]|e 3(e 3(e 3|t 3(e 3(e 3|( 3(e ^ 

* Documents acquired by ERIC include many informal unpublished • * 

* materials not available from other sources. ESIC makes every effort * 
*^ to obtain the best copy available. Nevertheless, items of mar.ginal * 

* reproducibility are often encountered and this affects the quality * 

* of the microfiche "^and hardcopy reproductions ERIC makes available * 

* via the ERIC Document Reproduction Service. (EDRS) • EDRS is not * 

* responsible for the quality of the original document* Reproductions * 

* supplied by EDRS are the best that can be made from the original. * 

3|e 4( 3|e 3|e 4( 3|e 4t « 3te 3te 4e 3te 3te 4e 3te sfe sfe i)t sfe sfe 4t 3te 3te 4e 3te 4e 4c 3|e 3(e 3^ 

/ ' 



/56I3. 



r-H' 
Q 
UJ 



EVALUATING COMPENSATORY EDUCATION PROGRAM TEST 
RESULTS USING EACH COMPENSATORY TEACHER'S 
PUPILS AS SUBGROUPS FOR ANALYSIS , 



U S OE^ARTMENTOFHEALTH 
EDUCATION 4 WELFAHE 
' NATlTbNAL INSTITUTEOF 
yEOUCATlON 

TMtS DOCuyENT MAS BEEN «£PRO 
?!i5l2J,r^^'-'' *^ R6CEIVIO FROM 
TM6 PEPSOM OR OftOANl2ATiON ORlOtN 
ATihjo IT POiMSOF VIEW Oft OPlNrONS*" 
STATED DO NOT NECESSARiiY «EPRE 
SENTOPPICIAL NATiONAl INSTITUTEOP 
EDUCATION POSrTiON OR POUCY 



^ Bureau of Evaluation and t due at tonal Seme es 

SX) Divisfon of Instructional Services 

^ State Department of Education 

/ 6 5 Capitol A venue 
<:> ' Bo\ 2219 

Q Ikrtford, Connecticut 0611 S 



ERIC 



EVALUATING COMPENSATORY EDUCATION PROGRAM TEST 
RESULTS USING EACH COMPENSATORY TEACHER'S 
PUPILS A^ SUBGRpUPS FOR ANALYSIS 

/ ' Page 
Sec tion I Charac ter/^tK s of Thi \ ^ valuation Mode/ 

Rationale 1 

Implemenlalion Con^»ideralions 1 

Implemenialion Procedures 1 

Analyzing the Data ' 2 

Reporting the Results 3 

MAT Gains Tables ' 4 

Sectidn II. Typical Worksheets That Could Be Used 

Shep Sl Marshall's PROJECT PUPILS TEST RECORD FORM .... 6 

Casper's PROJECT PUPIL'S TEST RECORD FORM . . . ^ . . . . . 8 
Follow Through Pupil Achievemenl 

and Influencing Factors • 9 

Reading & Malh Program Achievement 

and Influencing Factors . \ . , ^ . . . . 10 

Section 111' The Written Report and Attachments 

' Body of the Report 11 

Attachment 1 Distribution of Pupil's Raw Scores 13 

Attachment 2. &roup Scores for Reading and Math 13 

Attachment 3 Individual Pupil Data (Shep & Marshall) 14 

Attachment 4.r Individual Pupil Data (Casper) 14 

^ January 1975 . 



Bureau ot L valuation and I dm adonal Servu es 
biyision ot InstriK tional Setvnes 
Stoic Depa/tment of I duiation 
^ /65 Capitol Aienue 

^ Box 2210 

Hat tf Old, Connei tii^ut 0(yl 1^ 

ERIC \ 3 



EVALUATING COMPENSATORY EDUCATION PROGRAM TEST 
RESULTS USING EACH COMPENSATORY TEACHER'S 
PUPILS AS SUBGROUPS FOR ANALYSIS 

Rationale , 

A bai>ic decision in evaluating a program is the division of pupils into analysis 
subgroups. Because uf the advantages of having large numbers o'f pupils in an 
analy sis, there is a temptation to analyse all available project children in as large 
a group as possible This practice is not justified when distinct subgroups of pu- 
pils are represented 

The smallest identifiable subgroup? in compensatory education programs 
suitable for evaluation purposes are the projetl pupils of eath compensatory 
teacher, aide, or teacher aide team. Sample sizes are usually large enough gener- 
ally averaging from 1 5 io^O pupils The samples are alway s atv pical in that pu- 
pils are selected becauise they are experiencing problem> in stlftol work. Reading 
is the compensatory help most often provided with math second most prevalent. 
In >ome tases, a teacher provides reading to some pupils and math to others with- 
in the >ame group. Often, the compensatory leather is assigned pupils from more 
than one grade level. 

Any evaluation model must fit the above described circumstances. Its primary 
purpose should be to determine whether a compensatory leather's pupils achieve 
as they shoujd in reading or math. One approach is to determine the lest score 
gain a pupil a^^hieves from the beginning to the end of the year. All of the pupil 
gain Scores of a single compensatory teacher can be compared to those presented 
m the MAT Gains Tables. 

The MAT Gain's Tables presents typioal gams that should be made on a par- 
titular subtest of the Metropolitan Achiev^fQjMat Tests according to the grade 
level of the pupil and according to whether the pupil's pretest achievement level 
is high, average, or low. It differ^ from the norm tables provided in the test pub- 
lisher's manual m that both the pre and the posttest vvcre administered to the 
same pupils to obtain the gain scores. 

It has two major limitations. First, it is only useful for pupils' test results who 
are in grade fevels 2 8. Secondly , the interval between testing is short. 
Jmplementation Considerations' 

This evaluation model requires the use^ of ihc Reading, Math Computation, or 
Math Concepts subtests of the 1 970 edition of the Metropolitan Achievement 
Test, or, the corresponding subtests of any other standardized tea which can be 
converted into the Metropolitan. ^ 

The subtest shuuld be administered in October and in Apiii S"hc pupil should 
be administered the "level" ^ the test recommended for the pupil's grade level 
by Ihe test publisher i>r one level 1 >wer where pupil's skill proficiency in the test 
area is extremely luw. Different forms of the subtest should be used in October 
und April 

\ Fo Use thi^ evaluathjn model, project pupils must not be selected ti) receive 
compensatory help on-ffie basis of their October scores. 
Implementation PrtK cdures 

Step One SeUcl one ot the three mentioned ^.ubtest-) lor each pupil which is 
must clowly* I elated to instrLictii».i provided to^it pupil by ihe cunipensatt>ry 
leachej. Ihe pupil should be Jiitnimcd to ansvvn only those items on the test 
that he knows and to avoid guessing Administer and score the pretest inexact 



compliance wKh the procedures specified by ihc lest publisher. 

Step Two Record the raw stores for each pupil un the Project Pupils Test 
Record Form. Determine the stanine and Standard scon* fur each pupil's Octo- 
ber score using the norm tables in the Metropolitan Teacher's Manual. 

Step Three In April determj^nc the representative sample of ihe teacher's 
pupils whose scores will be submitted on the Individual Pupil Information 
Forms. Then administer the posttcst and ugain advise the pupil tu avuid guessing.- 
Scure and record the raw score and standard score fur each pupTI on the Project 
Pupils Test Record Form. 

Step Four. Subtract the Octuber standard score from the April standard 
score and record the gatn scure on the Prefect Pupils Test Record Form. Then , 
refer tu each pupil's Octuber stanine to determine whichxolumn uf the MAT 
Gains Table should be used. For stanines 1-3, use the LOW PRETEST column. 
Fo/stanines4-6, use the AVERAGE PRETEST column. And for stanine 7-9, 
useXhe HIGH PRETEST column. If the pupil's standard score gain equals ur 
e^ifceds the mean gain scure presented in the apprupriate column of the MAT 
Gams Table, a should be recorded in the last colunnn un the Project Pupils 
Test Record Form. If the pupil's pre to post test gain score was less th<in the 
mean score presented in the MAT Gams Tables, a should be recorded. Pro 
ceed m this manner until a " + " or a " has been recorded for each project pu- 
pil for whom preMnd posttest scores^are available. 

Step Five Determine the distribution uf pupils' raw scores for each level of 
each subtest administered. 
Analyzing the Data ' 

Interpret the pupils' raw score distributions for the October and April test 
' administrations. 

Determine the pruportiun uf the teacher's pupils whu achieved in reading or 
math as they shuuld^have by dividing the number of "pluses" by the number of 
"pluses and minuses," A mean propurtiun of .63 was found for 111 Cunnecticut 
cumpensatury teachers of reading and math in 1973 74, This standard can be 
used to compare a compensatory teacher's pupils with in 1974-75. 

Try to discover why some pupils achieved as they shuuld have while others 
did not. Fur example, compare the two jiroupings in terms of such factors as the 
fullowing (1) average grade level fur each grouping, (2) the pruportiun of buys 
cumpared tu girls making up each grouping, (3) an average pretest stanine cum- 
parison, (4) any" differences in the time uf day pupils were scheduled for help or 
the concentratiun ^^f compensato/y help providccJ, ur (5) the averagiV absences 
, from schoul for the two gruupings, Comparisuns such as these iiindjo identify 
strengths and weJkncsscs of compensatory prugramming, or, limiy.tiuns of the 
evaluation model used. 

A figure showing the distribution of pupils' ravv scores will be v<^|Cable in any 
test Score analysis, h will indicate whether there is a floor ur ceilingVffect ur 
whether scores "bunch up " Any of these charictcristics limit the usefulness of 
the tc^l results obtained and should be acknowledged as a limitation in the 
evaluation report. 

The distribution ot scores also shows the number of pupils scoring si^low 
that results tould be due mainly to guessing. Ftfr example, the Intermediate 
Level of the Metropolitan Reading test is a ICsl of 45 fi)ur-choice items which 
yields a mean chance score of one -fourth of the total nurriber of items, or 11.3 



When a puptl scores this low, it is difficult tu diy-crn whether he actually knuws 
the Items uf the test ur guessed them. Extremely luw turrelatiuns between fall 
and spring test results are mainly due to guessing and this behavioir nullifies the 
usefulness of tests as evaluation instruments 
Reporting the Results 

The year-end tumpensaturv pr^ogram evaluation report should start with a 
brief description of the evaluation model used. Follow ing this, the appropriate- 
ness of the Levels of subtests administered in terms of the distribu lion of raw 
scores should be discussed. Next, the number of each tompen^atory teacher's 
pupils who achieved as they should have followed by the number of children 
who did not st^ould be listed. The final part of the analysis should discuss the 
various factors that might bear on the test results obtained and their possible in- 
fluence upon how well the pupils achieved. ^ 

A table should appear in the appendix of the ref^^rt mdi^ating'^tte exact 
name of the instrument used, the subtest(s), levels, forms, ^iates of test adminis- 
tration, and pre and posttesl raw scores by grade levels of pupils. Combined 
results for all cocnpensalorVv efforts of a single project should-appear in this 
table. 

l-ndividual Pupil Reading or* Math Information Forms completed by each 
compensatory supported teacher should also appear in the appendix. These will 
be used for a statewide and rjational evaluation of compensatory education. 

MAT GAINS TABLETS 

Twenty of th^ school systems in the standardization of the 1970Metropol 
itan Achievement Tests provided data referred to as the MA TGatns Tables in 
this publication. These 20 systems were selected to be Representative of the en- 
tire standardization group (and thus, of the nation's school population) in terms 
of relevant population characteristics. 

Average Otis-Lennon Mental Ability Test deviation IQs for this sample ranged 
from a low of 99.6 (Grade 8) to a^igh of 101 .1 (Grade 5). Key variables used 
in selecting and describing the Metropolitan sample included the median years 
of schooling of adults over age 25 in the community , median family income, 
and the percent of blacks in the population. 

The data presented here have important advantages over "growth" charts or ^ 
tables offered in the past.'First, the data are empirical no interpolation or 
extrapolations are involved. Second, and perhaps more importantly, the same 
pupils were used for'^computing the Fall-Spring score changes. The regular 
MctropoliVtn percentile rank/stanine tables provide the first advantage above. 
However, the regular "Beginning^ and" "End" c>f y ear norms are not based on 
identical sets of pupils, although great care was taken to match the two samples 
as closely as possible. 

An additional advantage of these data is that tfie sample is closely reprCscata" 
live ol bulh the entire Met/upolilan normative sample andahe nation's school 
popufatioii, thus making interpretation of obtained results more meaningful. 

1 1 Is recognized that u lilization ot this mt>del is probably .loo lengthy a 
pruccss where the number ot staff in the project becomes veiy Urge. Add- 
itionally, the model is suitable t^nly for chifdren ^ho are fluent in English. 



MAT GAINS TABLES 
Median, Mean and S.D. of MAT Standard Score "Gain/* Over a Six-Monlh 
Period by Grade for Three Subgroups and Tolal £ruup (N= 146 1-2861 per grade) 
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ANALYSIS OF TEST RESULTS FOR THE MUNSEN SCHOOL 
COMPENSATORY IpUCATION PROGRAM 

One of two sublebti> of the Metropolitan Achievement Tests, Reading or Math 
Computation, wdb adminiblered to pupib in the Munsen Compensatory Program 
In October 1974 and April 1975. 

The teslicore gain ee^h pupil made wai^ compared to an "expected gain" for 
the particular subtest according to the grade level of the pupi! and according to 
whether the pupil's pretest achievement was high, average, or low. "Expected 
gains" were determined from the MAT Gains Tables." 

This test analysis is for three major purposes.. (1 ) to Interpret the distribu- 
tion of raw scores for the different levels of the tqsts administered, (2) to deter 
mine ^he number of a compensator> teacher's pupi^ who achieved as th^ 
should ha\^p and, the nurnber who did not, and {3) lo discuss factors which 
possibly influenced the pupils' test gains. • • 
. Pupils have been divided into two subgroups fpr test score analysis. (1) the 
reading and math results of middle grade pupils all served by the same compen . 
sdtory teacher, and (2) the reading results f9r grade 2 and:? pupifs re'teiving the 
Follow Through services. This latter , group is served by two aides who have 
overlapping responsibilities for the grade 2 and 3 pupils. 

A^distnbution of the raw scores for each level of each subtest was made. 
Reading Ust scores for grade 3- pupils showed some "piling up" of scores at the 
upper end of the scale at ppsttesting indicating that a higher level of that par 
ticular subtest should have been used as the test publisher/ecommended. 

The Reading pretest administered to grade 4, 5, and 6 pupils'should have 
been one level lower than that recommended by the test publisher as these pU 
pils all tended to score too low on the scale. 

When children score very low on a test^ it is difficult to ascertain whether 
the results are actually what the pupil knows or whether the fesults are due to^ 
guessing. , * " . * 

The raw score distributions sljow a third characteristic, that approximately 
^^three-quarters of the grade 3 pupils are achieving exceedingly wck All these pu 
pils began m the Auburn Headstart Programl^ve years earlier and riave been 
provided supplementary services each year sin£e. ^ . ^ 

Scores were not available for 1 2 children. Four of the pupils left Mpnscn 
School before the April testing, five entered Munsep School after the October 
testing, and three pupils received short term help unrelatedjo reading or math , 
^computation. 

The number of Children who achieved well and those who did not achieve u{?* 
lo expectation are presented below, . 

Compensatory Pupils' Achievement Follow Through, Reai^li^gand Maih 
by Teacher Subgroups: Aide Team Teacher 

Tptatpupils assigned for connipensaiory 38 21 

inslruclion during the 1974-75 year » \ t 

Total pupils for ^yhom pre and posiicsi * 29 18 

results have been reported ' , 

..Ndrnbtjr of pupils who achieved up to .^16 ^0 

expectations in readi/Sg or math ' . .''^ 

J^"mber of pupils who;didc^;o/ achieve up 13 , 8 

"tnexpectation In reading or taalh * * * - . ' / II 



These results indicate that slightly more than half of the pupils of each 
compensatory teachejc achieved in reading or math as well asjhey should have. 
This proportion is slightly less than the average proportion (X= ,63) obtained 
for 1 1 1 reading and math teachers of compensatory pupils in Connecticut in ' 
1973-74. 

In a further analysis, Various information was analyzed m an attetnpt tode 
termine factors possibly responsible for the differences in achievement found 
for the two categories 6f pupils. 

Follow Through Component Analysis 

Findings. There were^essentially no differences between {he two groups in 
terms of boy girl ratio, and pupil school abserfc6s. A slightly larger propor- 
tion of grade ^ pupils achieved as expected than did gradp 2 pupils. 

Pretest scpres were generally lower for pupils in tJie group who made the 
expected achievement. Also, these pupils show'ed the greater reading .test 
average gain from pre to posttesting (1 2 standard score gain units for the 
better, achievers compared to 2 standard scpre,^ain units for the others). 

Interjf>retati6n. Pupilsrhaving the greatest eclucational need showed the * 
better reading achie?vement. Other factors ex*amined appearecJHp have no 
particular influence on wJwlher th^ pupils achieved well or not. 

If more children are expected to achieve well in the Follow Through Pro- 
grarti in the coming year, it would be worthwhile for classroom teachers and 
compensatory aides "to review the methods and procedures they use to help 
their average and above average achieving pupils. 

Findings. There were no grade level differences befVveen groups. A larger 
proportion of boys than girls achieved as they should have. Pre test scores . 
were lower for pupils in the group who made expected achievement, and this 
group made the greater gains on the average (19 standard score units fo^the 
, better achievers compared to 4 standard score upits for the group not 
achieving as expected). Also, pupil absences from school appeared to be less 
for the group achieving as expected. * ' 

Interpretation. Again the pupils having the greatest educational need 
showed the be^tter reading and math, progress. 

Three factors should bte studied if more pupils are to achieve as they 
should in this program in the coming year: v 

Why do a greater proportion of boys than girls makp the expected 
achievement gains? 

How can schbol attendance of some project children be improved? 
How can achievement be improved for children who are a notch or two 
higher than the pupils who st^rt out lowest in the project group? 
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